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Abstract: This paper investigates an open issue related to false discovery rate (FDR) 
control of step-up-down (SUD) multiple testing procedures. It has been established 
in earlier literature that for this type of procedure, under some broad conditions, 
and in an asymptotical sense, the FDR is maximum when the signal strength under 
the alternative is maximum. In other words, so-called "Dirac uniform configura- 
tions" are asymptotically least favorable in this setting. It is known that this prop- 
erty also holds in a non-asymptotical sense (for any finite number of hypotheses), 
for the two extreme versions of SUD procedures, namely step-up and step-down 
(with extra conditions for the step-down case). It is therefore very natural to con- 
jecture that this non-asymptotical least favorable configuration property could more 
generally be true for all "intermediate" forms of SUD procedures. We prove that 
this is, somewhat surprisingly, not the case. The argument is based on the exact 



calculations proposed earlier by Roquain and Villers (2011al, that we extend here 



by generalizing Steck's recursion to the case of two populations. Secondly, we quan- 
tify the magnitude of this phenomenon by providing a nonasymptotic upper-bound 
and explicit vanishing rates as a function of the total number of hypotheses. 

Key words and phrases: False discovery rate, least favorable configuration, multiple 
testing, Steck's recursions, step-up-down. 



1 Introduction 

In mathematical statistics, so-called least favorable parameter configurations 
(LFCs) play a pivotal role. For a given statistical decision problem over a param- 
eter space and a given decision rule 6, we define an LFC 6* {5) as any element 
of that maximizes the risk (expected loss) of 5 under this parameter, i.e., 

ye G e, R{e,5) < R{e*{6),6), 



where R{9,5) denotes the risk of rule 5 under 6. When available, the knowledge 
of an LFCs allows one to obtain a bound on the risk over a possibly very large 
parameter space, including non- or semi-parametric cases where Q has infinite 
dimensionality. Theoretical investigations of minimax properties can rely on the 
computation of an LFC. Such knowledge is also relevant for practice, because a 
user of the procedure can be provided with a performance guarantee if an LFC 
is known. In this case, even if the risk under the LFC cannot be computed 
in closed form, it can be approximated by a Monte-Carlo method simulating 
the distribution corresponding to the LFC. Finally, if the parameter space 
is partitioned into disjoint, restricted submodels, it can be of interest to gain 
knowledge of the LFC of a decision rule 6 separately over each submodel, thus 
providing finer-grained information. 

LFC considerations naturally occur in hypothesis testing problems. A clas- 
sical example is that of one-sided tests over a one-dimensional parameter space 
admitting an isotonic likelihood ratio: it is well-known that the LFC for the type 
I error probability is located at the boundary of the null hypothesis. This fact is 
used to derive critical values for uniformly more powerful tests in that setting. 

The LFC problem is particularly delicate for multiple hypotheses testing and 



the latter has been investigated by many authors in previous literature (Finner 



and Roters, 2001: Benjamini and Yekutieli, 2001: Lehmann and Romano 2005 



Finner, Dickhaus, and Roters 2007: Romano and Wolf, 2007: Quo and Rao 



2008 Somerville and Hemmelmann 2008 Finner, Dickhaus, and Roters, 2009 



Finner and Gontscharuk, 2009, Gontscharuk, 2010). In that setting, a family of 



m > 2 null hypotheses Hi, ... , Hm is to be tested simultaneously under the scope 
of a common statistical model with parameter space 0, and some type I error 
criterion is used that accounts for multiplicity. For theoretical as well as practi- 
cal applications, it is relevant to determine LFCs over the restricted parameter 
spaces 0m,mo where exactly mo out of m of the null hypotheses are true. In this 
setting, LFC results can be derived straightforwardly only in special situations. 
In the present work, we restrict our attention to multiple testing procedures that 
depend on the observed data only through a collection of marginal p- values, each 
associated to an individual null hypothesis. This is a commonly used setting for 
multiple testing problems in high dimension. Moreover, we consider procedures 
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that reject exactly those nuh hypotheses having their p-value less than a certain 
common threshold t* , which can possibly be data-dependent. That is, t* may 
depend in a complex way of the entire family of p- values. We call such procedures 
threshold-based for short. 

In this setting, LFCs crucially depend on the type I error criterion considered. 
One frequently encountered family of such criteria is given through loss functions 
that only depend on the number of type I errors, denoted 

Vm = Vra{0, 5) := \{1 < i < ui : Hi is true for 6 and gets rejected by 6}\. (1) 

In other words, the risk takes the form R{6,5) = E,g[(f)(Vm)]- Natural assump- 
tions are that t* is a nonincreasing function in each p- value and that is a non- 
decreasing function. Then, by additionally assuming that the p-values are jointly 
independent, it is known that the LFC over 0m, mo is a Dirac-uniform (DU) dis- 



tribution (introduced by Finner and Roters, 2001), i.e. , such that values cor 



responding to true nulls are independent uniform variables, while p- values under 
alternatives follow a Dirac distribution with point mass 1 in zero. This result 
is formally restated in Appendix [C| For example, this LFC property holds true 
under the above assumptions for the /c-family-wise error rate (/c-FWER). For a 
given 6* G e, the /c-FWER under d is defined by FWERfc,^ := Fg{Vm > k). Strong 
control of the (l-)family-wise error rate, i. e., ensuring that supg FWERi g < a 
for a pre-defined level a G (0, 1), is the usual type I error concept in traditional 
multiple hypotheses testing theory. 

However, over the last two decades, progress in application fields such as 
genomics, proteomics, neuroimaging, and astronomy has lead to massive multi- 



ple testing problems with very large systems of hypotheses ( Dudoit and van der 



|Laan[ 12008] IPantazis, Nichols, Baillet, and Leahy[|2005] [Miller, Genovese, Nichol, 



Wasserman, Connolly, Reichart, Hopkins, Schneider, and Moore 2001). In this 



type of applications, (A:-)FWER control is typically too strict a requirement, and 
a less stringent notion of type I error control is needed in order to ensure reason- 
able power of corresponding multiple tests. In particular, the false discovery rate 



(FDR) introduced by Benjamini and Hochberg (1995) has become a standard 



criterion for type I error control in large-scale multiple testing problems. The 
FDR is defined as the expected proportion of type I errors among all rejections. 
Unfortunately, it does not fall into the class of type I error measures considered 
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in the previous paragraph, so that the above result does not apply. Furthermore, 
because the average ratio of two dependent random variables is not necessarily 
increasing in the value of the numerator, the LFC problem for the FDR criterion 
turns out to be a challenging issue - even for simple classes of multiple tests and 
under independence assumptions. 

In this work, we contribute to the theory of LFCs under the FDR criterion 
for so-called step-up-down multiple tests (SUD procedures, for short). These pro- 
cedures constitute an important and wide subclass of threshold-based multiple 
testing procedures, wherein the threshold t* is obtained by comparing the re- 



ordered p- values to a fixed set of critical values (see Tamhane, Liu, and Dunnett 



1998 Sarkar, 2002). Furthermore, recent research has reinforced the interest 



of this type of procedures. For instance, Finner et al. (2009) have shown that 
step-up-down tests can be used is association with the so-called asymptotically 
optimal rejection curve (AORC) to provide an asymptotically (as m — )• oo) valid 
FDR control which is additionally optimal in some specific sense. 
Namely, the contributions of the paper are as follows: 

• a survey of known LFC results for SUD procedures in specific model classes 
is provided in Section [3j 

• new results and surprising counterexamples for LFCs of SUD procedures 
are derived in Section HI 

• in Section [5} Steck's recursion is extended to the case of two populations 
and we provide a summary of the exact formulas for computing the FDR 
proposed by Roquain and Villers ( 2011a|b ); these formulas are used to 
derive the counterexamples previously mentioned. 



2 Mathematical setting 
2.1 Models 

Given a statistical model, we consider a finite set of m > 2 null hypotheses 
Hi, . . . , Hm, and a corresponding, fixed collection of tests with associated p- 
value family p := {pi,i £ {1, m}). For simplicity, we skip somewhat the formal 
definition of p- values and of the underlying statistical model and consider directly 
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a statistical model for the p-values, that is, a model for the joint distribution of 
p. In what follows, we denote by J- the set containing c.d.f.'s from [0,1] into 
[0, 1] that are continuous. 

• The value family p follows the (two group) fixed mixture model with pa- 
rameters m > 2, 1 < rriQ < m and F G J-", for which the corresponding 
distribution is denoted by FM(m,mo,-F), if p = {pi,i G {!,..., m}) is a 
family of mutually independent variables and for all i, 



Pi 



?7(0, 1) ifl<i<mo, 

F if rriQ + 1 < i < m, 



where C/(0, 1) denotes the uniform distribution on (0, 1). 

• The p- value family p follows the (two group) random mixture model with 
parameters m > 2, ttq G [0, 1] and F £ J^, for which the corresponding 
distribution is denoted by RM(m, ttq, -F), if there is an (unobserved) bino- 
mial random variable mo ~ B{m, ttq) such that p follows the FM(m, mo, F) 
model conditionally on mo- In that case, the p- values are i.i.d. with (un- 
conditional) c.d.f. G{t) = TTot + (1 — 7ro)F(t). 

In the above definition, note that the true nulls are automatically assigned 
to the mo (random or not) first coordinates. This can be assumed without loss 
of generality, since we only consider procedures which only depend on p-values 
through their reordering in increasing order. 

A common additional assumption on F is that F{x) > x or that F is concave. 
For instance, these assumptions are both satisfied in the two following standard 
examples: 

- Gaussian location model: F{t) = <!>($ ^(t) — ^), for a given alternative 
mean ^ > 0, where ^{z) = ¥{Z > z) for Z ~ A/'(0, 1). This corresponds 
to the alternative distribution of p- values when testing for ^ < under a 
Gaussian location shift model with unit variance. 



Dirac 6q distribution: F is identically equal to 1, as introduced by Finner 



and Roters (2001). The corresponding distribution in the FM model is 



called Dirac-uniform (DU) configuration (or distribution) and denoted by 
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FM(m, rn-o, -F = 1) or simply DU(m, mo). We define similarly RM(m, vro, F = 
1). Note that the Dirac-uniform configuration can be seen as an instance 
of the Gaussian c.d.f. for an alternative mean = co. 

In the existing literature, the Dirac-uniform distribution has often be considered 
as the first candidate for being an LFC of several global type I error rates (with or 



without a theoretical support) (see, e.g., Finner et al. , 2007, Romano and Wolf 



2007 Somerville and Hemmelmann, 2008). 



2.2 Procedures 

In this paper, we consider the particular class of multiple testing procedures called 



step- up-down procedures, introduced by Tamhane et al. (1998), see also Sarkar 



(2002). First define a threshold or critical value collection as any nondecreasing 
sequence t = (tfc)i<A,<m £ [0, 1]™ (with to = by convention). 

Definition 2.1. Let us order the p-values < p(2) ^ ••• ^ P{m) (with the 
convention p(o) = 0). For any threshold collection t, the step-up-down (SUD) 
procedure with threshold collection t and of order A G {1, ...,m}, denoted here by 
SUDx{t), rejects the i-th hypothesis if pi < tj,, with 



ifPix) > tx- 



I maxjfc G {A, . . . , m} : V/c' G {A, . 
k = < 

[ ma^{k G {0, . . . , A} : p(^k) < 4} 

In the sequel, for convenience, we identify procedures with their rejection 
sets, e.g., SUD;v(t) = {1 < i < m : Pi < t^}. An important remark is that 
the cases A = 1 and X = m correspond to the traditional step-down and step-up 
procedures, respectively. An illustration is provided in Figure [TJ 



A classical choice for the threshold collection t consists of Simes' ( 1986 ) crit- 
ical values tk = ak/m for a pre-specified level a G (0,1). The corresponding 
step-up-down procedure is called the linear step-up-down procedure and is de- 
noted by LSUDa. In particular, for A = 1 and A = m, the procedure LSUDa is 
simply denoted by LSD and LSU, respectively. LSU corresponds to the famous 



linear step-up procedure of Benjamini and Hochberg (1995). 



It is common to consider threshold collections of the form tk = p{k/m) for 
a function p : [0,1] — )• [0,1]. This function is generally assumed to satisfy the 
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Figure 1: Value of k (vertical solid line), defined by for several procedures of the 
SUD type. The bottom-right SUD procedure (using A — 2m/3) coincides with the SU 
procedure for this realization of the p-value family. 



following assumptions: 

p : [0, 1] —7- [0, 1] is continuous and non-decreasing; (3) 
X £ (0, 1] — p{x)/x is non-decreasing. (4) 

The function p is called the critical value function (while its inverse is generally 



called the rejection curve, see e.g. Finner et al. , 2009). Observe that assumption 



([3]) can always be assumed to hold when m is fixed, however it is of interest in 
the case of an asymptotical analysis as m — )■ 00 (in which case p is assumed to 
be independent of m). Assumption (Q on the other hand restricts the possible 
threshold collection also for any fixed m. It will be often used in this paper. 
For a fixed finite m, assumptions ([s]) and (|4]) taken together are equivalent to 
"A; I— )• tk/k is non-decreasing". 
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2.3 False discovery rate and LFCs 



As introduced by Benjamini and Hochberg (1995), the false discovery rate of 



a multiple testing procedure is defined as the averaged ratio of the number of 
erroneous rejections to the total number of rejections. In our setting, for a 
distribution P being either FM(m,mo,-F) or RM(m, vro, F), the FDR of a step- 
up-down procedure can be written as 

FDR(SUDA(t), P) = Ep^p [FDP(SUDA(t), mo, p)] , (5) 

for which the FDP is the false discovery proportion defined by 

111 < ^ < mo : Pi < if 1 1 
FDP SUDa t , mo, p = ,1 - , 6 

where | • | denotes the cardinality function and in which mo is either fixed or ran- 
dom whether P is FM(m, mo,-F) or RM(m, ttq, F), respectively. For short, the 
quantity FDR(SUDA(t), FM(m, mo, F)) is often denoted FDR(SUDA(t), mo, F), 
or FDR(SUDA(t), F) when the context makes the interpretation unambiguous. 
Similarly, FDR(SUDA(t), RM(m, tto, F)) can be shortened as FDR(SUDA(t), tto, F), 
or FDR(SUDA(t),F). 

Definition 2.2. Any F' £ F is called a least favorable configuration (LFC) for 
the FDR of SUDx{t) in a fixed mixture model with tuq true hypotheses out of m 
and over the class F if 

VF G F, FDR{SUDx{t),mo,F) < FDR{SUDx{t),mo, F'). 

A similar definition holds for a random mixture model with m hypotheses and 
proportion ttq of true hypotheses. 

The above definition can possibly be restricted to a subclass F' C F (typi- 
cally, the class of concave c.d.f.s). This will be clearly specified in the context. 

Obviously, if F' is an LFC for the fixed mixture model for all values of 
mo, then it is also an LFC in the RM(m,7ro,-F) model for any value of vro (by 
integrating over mo ~ B{m, vro)). 
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3 Survey of known LFCs under the FDR criterion 



Recent results about LFCs for the FDR criterion related to step-up-down type 
procedures are summarized in Figure [2] and explained below. These results hold 
either under the fixed mixture model or the random mixture model, hence involve 
a maximization over distributions where the p- values are independent. (While 
the present paper is focused on this setting, let us mention here briefly, that LFCs 
for the FDR criterion under arbitrary dependencies have also been studied, see, 
e.g. 



Lehmann and Romano 2005; Quo and Rao, 2008 



First, let us consider the problem of the monotonicity of FDR(SUDA(t)) in A 
(vertical arrows). Recently, it was proved that, whenever F is concave, the FDR 
grows as the rejection set grows (Zeisel, Zuk, and Domany, 2011, Theorem 4.1). 
Interestingly, the rejection set R can have a very general form: the only condition 
is that is a measurable function of the order statistics of the family of p- values 
under consideration. From Q and since for any A G {1, . . . , m— 1}, the rejection 
set of SUD;^(t) is included in the one of SUD;^+i(t), we obtain that for a concave 
F, 

FDR(SUDA(t)) < FDR(SUDA+i(t)), 

both for FM(m,mo,-F) and RM(m, ttq, -F) models. This implies in particular 
that FDR(SD(t)) < FDR(SU(t)) for a concave F. Other studies estabhsh sim- 
ilar inequalities, but with a condition on the threshold collection t, not on F. 



Precisely, Theorem 4.3 of Finner et al. (2009) and Theorem 3.10 of Gontscharuk 



(2010) establish that, when k i— )■ tk/k is nondecreasing, for any A € {1, . . . , m— 1}, 

FDR(SUDA(t)) < FDR(SU(t)), 

both for FM{m,mQ, F) and RM(m, 7ro,-F) models. In particular, the fact that 
FDR(SU(t)) dominates the FDR of SD(t) is quite well established in multiple 
testing literature. Nevertheless, let us stress that this is no longer the case for 
"atypical" configurations of F and t, as we state in Appendix [B| 

Secondly, let us consider the monotonicity of FDR(SUDA(t), F) in F. In the 
step-up case (i.e., A = m), the situation is somewhat simple: Theorem 5.3 of 



Benjamini and Yekutieh (2001) states that F < F' implies FDR(SUDA(t), F) < 



FDR(SUDA(t), i^') whenever k i— )• tk/k is nondecreasing. Moreover, the inequal- 
ity is reversed whenever k ^ tk/k is nonincreasing. In the step-down case (i.e.. 



9 



A 



1) and for a RM(m, 7ro,-F) model, Theorem 4.1 of Roquain and Villers 



(2011a) states that the Dirac-uniform configuration (F = 1) is an LFC under 
some complex condition on the threshold collection t, that is fulfilled by the 
linear threshold collection = ak/m, a € (0, 1) and over the class of concave 
c.d.f.'s. However, for A ^ {l,m} (i.e., an "intermediate" SUD procedure), finding 
LFC's is more delicate and the only known result is asymptotic, as m tends to 



infinity. Precisely, combining Theorem 4.3 of Finner et al. (2009 ) and Lemma 3.7 



of Gontscharuk (2010), we easily derive the following result: 

Theorem 3.1. iGontscharuk (2010)] Consider a step-up-down procedure using 
a threshold collection of the form tk = p{k/m), where p satisfies ^ and (Q. 
Assume that the step-up-down procedure is performed at an order A = Am such 
that Xm/mn — >■ n G [0,1]- Assume that rriQ/m [0,1] o-nd that, under 

the DU(m,mo) distribution, the number of rejections of SUD\{t) satisfies that 
\SUD\[t)\/m converges in probability as m grows to infinity. Then, in the fixed 
mixture model FM(m, mo, F), we have for any F £ J^, 



limsup {FDR{SUDxJt),F) - FDR{SUDxJt), F = 1)} < 0, (7) 

m 

either for all ( e [0,1] if k > or for all ( S [0, 1) if k = 0. 

However, for a finite m, and A ^ {l,m} no result is known about LFC's to 
our knowledge. This is the point of the paper and is symbolized by the question 
mark in the middle of Figure [2j 

Finally, let us consider the linear SUD procedure, that is, the SUD procedure 
using the threshold collection t^ = ak/m, a € (0,1) (corresponding to p{x) = 
ax). Since both LSU and LSD procedures satisfy that DU is an LFC and since 
an SUD procedure can be expressed as a combination of an SU and an SD, we 
might make the following conjecture, which is the starting point of this paper. 

Conjecture 3.2. For any m > 2, the Dirac-uniform configuration (F = 1) is 
a least favorable configuration for the FDR of the linear step-up-down procedure, 
in the IiM(m,7rQ, F) and FM{m,mQ, F) models. 

Obviously, a similar conjecture might be formulated for a (non- linear) step- 
up-down procedure using p satisfying ^ and (Q. 
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[RV2011] 
tk linear 
RM, F concave 




[ZZD2010] 
F concave 



[ZZD2010] 



- {SUD,^ H ^SUD,DU 



[FDR2009] 
[Gont2010] 
tk/k nondecreasing 



[ZZD201 0] 
F concave 



[ZZD2010] 



SU, DU 



[BY2001] 
tk/k nondecreasing 



Figure 2: An arrow "A B" means "FDR(^) < FDR(5)" . These results hold for the 
fixed mixture (FM) model except when "RM" is written. The brackets are a shortened 
reference to the corresponding literature, see main text for more details. 
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4 Investigating Conjecture 



3.2 



4.1 Disproving the conjecture 

The exact calculations described in Section [5] allow to compute the value of 
FDR(LSUDA(t)) exactly. This shows the following (numerical) result. 

Proposition 4.1. For m = 10, consider the linear step-up-down procedure 
LSUDx at level a = 0.5 and for A € {4, 5, 6, 7}. Then, we have 

FDR{LSUDx, F) > FDR{LSUDx, F = l), (8) 

in either of the two following cases: 

• in the FM{m,mo, F) model, with mo = 7 and F{x) = x; 

• in the RM(m, 7ro,-F) model, with ttq = 7/10 and F(x) = x. 



This disproves Conjecture 3.2 and shows that finding the LFC for SUD is 



more difficult than for SU and SD separately. More generally, Figure |3] reports 
some obtained values in the Gaussian case F = $($ ^(•) — for different val- 
ues of the alternative mean /i. We observe that the result for FM(m, mo,-F) 
(left panel) or RM(m, 7ro,-F) (right panel) are qualitatively the same: there is a 
range of values for A G {2,...,m — 1} for which the FDR is larger for a smaller 
II. However, this phenomenon seems to decay when m becomes larger, see Fig- 
ure [3] for m = 100. Also, when a decreases, the phenomenon still occurs but 
its amplitude decays. To alleviate the concern that this somewhat unexpected 
phenomenon could be due to numerical inaccuracies in the computation of the ex- 
act formulas (which involve several nested recursions), the reported results were 
double-checked via extensive and independent Monte-Carlo simulations, which 
confirmed the validity of the reported curves. 



4.2 Nonasymptotic bound 

In the present section, we investigate the amplitude of the phenomenon observed 
in the previous section as a function of the number of hypotheses. In other words, 
we derive a more explicit and non-asymptotic version of the limit appearing in 
(fTl). For this, we consider a perturbation analysis of the SUD procedure as defined 
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Figure 3: The LFC of LSUD is not always DU. FDR(LSUDa) as a function of the order 
A G {1,...,to}. Left: fixed mixture; Right: random mixture. One sided Gaussian 
location model with parameter jj. a = 0.5. 

in Q, under the Dirac-uniform model, and when the empirical c.d.f. of the p- 
values is (^-close to the population c.d.f. (which happens with large probability). 
In order to state the result in a compact form, we first introduce the following 
notation for an SUD threshold in a continuous setting. 

Definition 4.2. Let p : [0,1] [0,1] satisfying For any non- decreasing 
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function G : [0, 1] — )■ [0, 1], and £ £ [0, 1], we define 

^^^^^^imin{ue[£,l] : G(p(n)) < n} ^f G{p{£)) > £; 

\max{u G [0,^] : G{p{u)) > u} if G{p{£)) < £. 

Observe that in the above definition, the infimum and supremum are well- 
defined since the considered sets are non-empty; using that G is non-decreasing, 
it can be seen that U{£,G) is a fixed point of the function G o p (so that the 
infimum is indeed a minimum and the supremum, a maximum). Unfortunately, 
the number of rejections k of the SUD procedure as defined in ^ does not 
always satisfy k/m = U{X/m,Gm) (because of the step-down part, see Figure [g] 



in Section 7.2). Nevertheless, the following lemma is proved in Section 7.2 



Lemma 4.3. With the above notation, if the threshold collection t is defined as 
tk = p{k/m), we have 

U{\/m, Gm) < k/m < U{X/m, (G™ + m"^) A 1), (10) 

where k is defined by ^ and Gm(x) := rn~^ ^1!^^\{pi < x} is the empirical 
c.d.f. of the p-values. 

We now state our main result. 

Theorem 4.4. Consider a threshold collection t of the form tk = p{k/m), where 
p : [0,1] — )• [0,1] satisfies ([s]) and Q. Let ( G (0,1) be an arbitrary constant. 
For 5 e (0, 1), define 

uj =UiX,{Gf^ + 6) Al) and = U{X, {Gf^ - 6) V 0), 

where G^^{x) := (1 — C) + (see Figure^for an illustration). Let us define 
the remaining term: for any y G (0, 1), 

e{6,m,C,y) := "^"^^^ "/^"^"^ + ^ ^_2™(.-,-i,)^(i-,/c)._ ^^^^ 
uj 1-C 

Then, for any F £ and X £ {1, . . . , m} the following holds. 

• In the FM(m, mo, F) model with < mo < m and v = maxfcg|„jj_x ,„Q|{|/i;/m- 
C|} G [0, 1], we have 

FDR(SUDx(t), mo,F)< FDRiSUDx (t) , mo , F = 1) + — e((^, m, C, z^) ; 

m 

(12) 
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In the RM(m, ttq, F) model with ttq = C; we have for any 7 G (0, 1), 

FDR{SUDx{t),TTo, F) < FDR{SUDx{t),TTo, F = 1) + irosid, m, (, 7) + 4.e-^rn{^-^/m)l 

(13) 
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Figure 4: Illustration for and in the case of the LSUD and the SUD based on 
AORC. Here the X-axis is on the "threshold scale" t = p{u). ( = 0.5; S = 0.03. The 
area between {G^^ — (5) V and {G^^ + 5) A 1 is displayed in gray. 



Theorem |4.4| is proved in Section 7.3, This result is illustrated in the two 
following examples. 



1. Let us first apply Theorem 4.4 in the particular case where p{x) = ax is the 
linear critical value function, see the left panel of Figure |4j In that case, we 



have uj = V and ut = A 1. Hence, 

2aS 



{p{u^)-p{uj))/ut < — 



+ 5' 



As a result, (12) and (13) hold by replacing {p{u'g) — p{u^ ))/'^'s 
inside the remaining term e{5, m, C,, y). 



15 



2. Second, for Q > a, let us use 



p{u) = au/{l - u{l - a)), that is, p~'^{t) = t/{a + t{l - a)). (14) 

The rejection curve p~^, displayed in the right panel of Figure |4j is called 
the "asymptotically optimal rejection curve" (AORC). It was introduced 



by Finner et al. (2009) for the purpose of improving the power of the linear 



critical value function. When applying Theorem 4.4 with this choice of 
p, the calculation of uj and depends on the position of the parameter 
X-m/m on [0, 1] and p{u^) — p{uj) may not vanish when 6 becomes small, see 
Figure |4j Fortunately, when p{\m/m) (dotted-long-dashed line) is smaller 
than the quantity p{vs) (dashed line), the two points p{uj) and p{uf) 
(dotted-dashed lines) are expected to be close as 5 becomes small and the 
bound given in Theorem 4.4 will vanish. The exact expressions of , 



and vs can be easily derived by solving the corresponding quadratic 
equations. For short, we only report the equivalent as 5 tends to 0: 



vs = l-5a/{Q-a) + 0{5^) 

^5 = 
= 



= (1 - 0/(1 - a) + - a) + 0{5^) 
={l- 0/(1 - a) - <5C/(C - a) + 0{5^). 



Since p'{{l - 0/(1 - a)) = Q/C^ we have p{ul) - p{ug) = 2a5/{C - "0 + 
0((5^). As a result, assuming C, > a and A/m < vs, we can derive that 
(12) and (13) hold and that quantity {p{ug) — p{uj))/u'^ is equivalent to 



2a(l-a) 
(C^-aC){l-C) 



5 (as 6 tends to zero) in the remaining term e{6, m, y). 



4.3 Convergence rate when m — )■ oo 



We can now use Theorem 4.4 in an asymptotic sense and for specific critical value 
functions, in order to obtain an explicit bound on the convergence rate of the 
limit appearing in ([7]). 

Corollary 4.5. Let a G (0, 1). Consider a threshold collection of the form tk = 
p[k/m), where p is either p[x) = ax (linear) or given by (14), that is, associated 



with the AORC. Consider the SUD procedure of threshold collection t and of order 
A = \m possibly depending on m. Consider either the FM{m,mQ, F) model with 
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1^0 = LCm'TT'J or RM(m, ttq, F) model with ttq = Cm, for some (rn G 1) possibly 
depending on m. Assume that ^ (log m) / m = o(l). For the AORC case, 



assume moreover 



■sj (log m) / m = 0(1). Then we have for any F ^ F , 



l-\m/m 

{FDR{SUD^^{t),F) - FDR{SUD,^it),F^ 1))^ = O ( ^^/^ 



(15) 



Corollary |4.5| is proved in Section |7.4| and is an easy consequence of Theo- 
rem 4.4 when taking 5 = 5m (and 7 = jm) suitably tending to zero. Assuming 
Cm > a is not an important restriction because when Cm < ct) controlling the 
FDR is a trivial problem: the procedure rejecting all the hypotheses has an FDR 
(and even an FDP) smaller than Cm ^ ct. 

While focusing on the linear and AORC rejection curve, the conclusion of 



Corollary |4.5| is substantially more informative than Theorem 3.1 it evaluates 



what is the order of the error when considering that the DU is an LFC of an SUD 



test. For Cm = C ^ (Oi 1) fixed with m, note that the rate of convergence in ( 15 ) is 
equal to the parametric rate, up to a logm factor. Furthermore, the constant in 
the O(-) can be derived explicitly by using the bound from the previous section. 
For Cm tending to 1 (not too quickly, "fairly" sparse case), the convergence rate 
is slower. 



As a counterpart, assumptions of Corollary 4.5 are more restrictive than 
those of Theorem |3.1[ In particular, they exclude the case where Cm tends to 
1 faster than (log m) /m ("highly" sparse case). This is a limitation of the 
methodology used to prove the nonasymptotic results. This problem may pos- 



sibly be fixed by adapting the proof of Lemma 3.7 of Gontscharuk (2010) to a 



nonasymptotic setting, but this falls outside of the intended scope of this paper. 

5 Exact formulas 



In this section, we gather some of the formulas derived by Roquain and Villers 



( 2011a| |b) to calculate the joint distribution of the number of false discoveries and 



the number of discoveries. Moreover, we complement this work by giving a new 
recursion that makes these formulas fully usable. These calculations are used to 



state Proposition 4.1 
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5.1 A new Steck's recursion 

For any k > and any threshold cohection t = (ti, . . . , t^), we denote 

^k{t) = ^kih, . . . , tfc) = P [f/(i) < ti, . . . , f/(fc) < tk] . (16) 

where (C/j)i<i<fc is a sequence of k variables i.i.d. uniform on (0, 1) and with the 
convention ^o(") = 1- In practice, quantity ( [l6| ) can be evaluated using standard 
Steck's recursion ^'^(t) = (tk)'' - Y.)zl [^Mtk - tj+if-i^ j{ti, . . .,tj) (Shorack 



and Wellner] |1986[ p. 366-369). 

Next, we generalize the latter to the case of two populations. Define for 
^ ko < k and any threshold collection t = {ti, . . . ,tk), 

^k,koAti^ ■ ■ ■ ^ik) =^ [U^i) < h, . . . ,U^k) <tk] , (17) 

where (C/j)i<j<fc is a sequence of k variables such that (t/i)i<j<fco are i.i.d. uni- 
form on (0,1), independently of {Ui)ko+i<i<k i-i-d. of c.d.f. F and with the 
convention ^'o,o,f(") = 1- The computation of ^k,ko,F is more difficult than ^f^. 
because it involves non i.i.d. variables. To our knowledge the existing formulas 



for computing ^k,ko,F have a complexity exponential with k (Glueck, Karimpour 



Fard, Mandel, Hunter, and Muller 2008). Here, we propose a substantially less 



complex computation, by generalizing Steck's recursions as follows. 
Proposition 5.1. The following recursion holds: for < kQ < k, 

^k,koAh, ...,tk) ={tkf'F{t,t-'^^ - i^') " 

jo<ko 
j-jo<k-ko 

X (tk - t,+i)'^«-^°(i^(tfc) - F(t,-+i))^-'=o-^'+^"M/,-,„,p(ti, . . .,tj), 

(18) 

with the convention 0*^ = 1. 



This is proved in Section 7.1 Note that the case k = ko reduces to the 



standard (one population) Steck's recursion. 



5.2 FDR formulas 



Using the ^'^'s and ^'fc.fco's, let us define the following useful quantities: for any 
threshold collection t = (tfc)i<fc<mi F £ T; for any ttq E [0, 1], k > 0, k < m. 
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< j < k, we let 

^m,7ro,F(t, k,j) 



k-j 



TriTTt-'itkyiFm'-^ 



X ^m-kil - G{tm), . . . , 1 - 



m,iTo,F 



{t,k,j) 



^\ r^-3\ik-j 



k-j 



vr; 



m—k 



X . . .,tk), 



(19) 



(20) 



where G{t) = vrot + (1 — 7ro)F{t). For any mo G {0, . . . ,m}, k > 0, k < m, 
j < nT'O, k — j < m — mo, we let 



Qm,mo,F{t,k,j) 



J 



k-j 



X ^ 



it,k,j) 



m—k,mo—j,F 

mo\ fm — niQ 
3 J\ k-j 



X ^k,j,F{tl, . ■ ■,tk), 



(1 — tm, • • • , 1 — tfc+i); (21) 

(1 - tfc+i)"^°-^'(l - F{tk+i))"°-''+' 

(22) 



where F{t) = 1 — F{1 — t). The following results have been proved by Roquain 
and VillersI ( |2011a|bD . 



Theorem 5.2 (Roquain and Villers (2011)). Consider any threshold collection t 
and the quantities defined by (19), (20), (21) and ( |22[ ). Then the following holds: 

(i) In the RM(m, 7ro,-F) model, for any ttq G [0, 1], -F G J^, < < m, 
0<j<k, 



Fi\Rn{l,...,mo}\=j,\R\=k) 



Vrn,-.,A^,k,j) forR = SU{t), 
Vm,^,,F{-^,k,j) forR = SD{t). 

(23) 



(ii) In the FM(m, mo, F) model, for any mo G {0, . . . , m}, F G J^, < k < m, 
V (A; — m + mo) < j < mo A k, 



P(|i?n{l,...,mo}| =j,\R\ = k) 



(t,fc,i) for R = suit), 
it,k,j) forR = SD{t). 

(24) 
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Classically, any step-up-down procedure can be written as a combination of 



a step-down and a step- up procedure ( Sarkar , 2002 ) : 



) if |SU((tA Atj)i<j<™)| < A, 

^ " \ SD((tA V t,)i<,<m) if |SD((tA V t,)i<j<m)\ > A. 

Moreover, since {|SU((tA A tj)i<j<m)| < A} = {p(A) > tA} and {|SD((tA V 
^i)i<i<"i)l ^ -^l = {PW — ^-^i cases in ([25]) form a partition of the 

probability space. This yields the following explicit FDR computations: 

Corollary 5.3. let X £ {1, ... , m} and consider any threshold collection t. Then 
the following holds: 

(i) In the model RM(m, ttq, F), for any tto G [0, 1], F £ T , we have 

A-l k . 

FDR{SUDx{t)) =J2J2 f ^-,.o,F(t A tx, k,j) 

k=l j=0 
m k 



+ IZEi^™.-o,F(tVtA,A:,j). (26) 



k=X j=0 

(ii) In the model FM(m, mo, -F), for any rriQ G {0, . . . ,m}, F £ T, we have 

A— 1 moAk 

{tAtx,k,j) 

k=l j=OV(fc— m+mo) 
m moAfc 

+ Yl f Qm,mo,F(tVtA,A:,j). (27) 

fe=A j=OV(fc— m+mo) 

In the model RM(m, ttq, F), it turns out that FDR(SUDA(t)) has an expres- 
sion only involving the ^'fcS, and not the ^k,j,F (Roquain and Villers 



2011a 



Section 5.2). Although it has a somewhat less intuitive form, it is better than 



(26) from a computational point of view. 



6 Discussion 

Our aim in this paper was to address the question "is the Dirac-uniform distribu- 
tion an LFC for an intermediate step-up-down procedure (that uses a standard 
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threshold collection)?" In a nutshell, the answer we found is "no, but almost". 
We provided a rigorous quantification of what "almost" means, using an alter- 



native approach to the asymptotic results of Gontscharuk (2010) that entails 
nonasymptotic bounds and explicit convergence rates. In practical situations, 
evaluating such bounds can allow to determine whether we can consider that the 
FDR is maximum when the signal strength is maximum. 

Returning to equations ^ and ([G]) , an additional question, particularly rele- 
vant in practice, is how appropriate it is to base the multiple type I error criterion 
solely on control of the expectation of the random variable FDP. We notice that 



Theorem 5.2 may also be used to study this issue by computing exactly the point 
mass function of the FDP under arbitrary configurations for the alternative, cf. 
Section |A] in the appendix. Based on this, we investigated to what extent the 
distribution of the FDP concentrates around its expectation for a simple Gaus- 
sian location model with parameter Figure [5] was obtained from these exact 
formulas for m = 100 and varying values of ttq and /i. Note that the unrealis- 
tically large choice of a = 1/2 has only been used for reasons of readability of 
the figures; similar plots also obtained when choosing a smaller (the variance of 
the FDP actually increases with smaller a, because this entails a smaller number 
of rejections). On inspection of these graphs, it becomes apparent that - even 
though joint independence of the values holds - the distribution of the FDP is 
not concentrated around the corresponding FDR in the following two situations: 
(i) The effect size is close to zero (weak signal) or (ii) the proportion vro of true 
null hypotheses is close to 1 (sparse signal). Thus, controlling the FDR alone 
does not guarantee a small FDP for a specific experiment at hand in these cases. 
For a well-defined dependency structure induced by exchangeable test statistics, 
theoretical arguments for m tending to infinity support the observation that the 



distribution of the FDP often does not degenerate in the limit, see (Finner et al. 



2007 Delattre and Roquain, 2011). For the jointly independent case and in the 



cases (i) or (ii) above, this phenomenon has not been theoretically studied to the 
best of our knowledge. The latter can possibly be investigated by extending the 



asymptotic approach of Neuvial ( 2008 ) to the case where /x and ttq are allowed 
to depend on m. 

Taking these considerations into account, control of the false discovery ex- 
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Figure 5: Exact probability P(FDP(LSU) G [i/50, {i + l)/50)) for < i < 50. The value 
of FDR(LSU) = TToa is displayed by the vertical dashed line. Random mixture model. 
a — 0.5; m — 100. One sided location Gaussian c.d.f. with parameter fj.. 



ceedance (i.e., of the probability that the FDP exceeds a given threshold) has 
recently been proposed in the literature (see, e.g. Farcomeni, 2008| for a review). 
Controlling the false discovery exceedance control again brings forth the question 
of the corresponding LFC: are Dirac- uniform configurations least favorable for, 
e.g., P(FDP(LSU) > x)? Some non-reported figures show that this is not the 
case for any x. Hence, finding LFCs for the false discovery exceedance stays an 
open avenue for future research. 
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7 Proofs 

7.1 Proof of Proposition |5.1 



We follow the proof of the regular Steck's recursion (Shorack and Wellner 1986 
p. 366-369). By using the convention ?7(o) = = and by considering the 
smallest j for which > tj+i, we can write 

IP(f/{fc) < tfc) - IP [U^i) <ti,..., U^k) < tk] 

k-2 

= ^P(Vz < j, Uli) < U, U^j+i) > tj+i, Uk < tk) 

j=0 
k-2 

j=0 Xc{l,...,k},\X\=j 

Hence, if U^i-x) denotes the i-th smallest member of the set {Ui,i G X}, we 
obtain 

nu^k) <tk)-¥ [[/(I) <h,..., u^k) < tk] 

k~2 

i=0 Xc{l,...,A:},|X|=i 
fc-2 j 

= J2Y1 E l{|^n{l,...,/co}| =jo}*,jo,F(ti,...,ii) 

i=o jo=o xc{i,...,fc},|x|=j 

xP(Vi^X,tj+i<C/i<tfc). 
7.2 Proof of Lemma 14.31 

In this proof, we denote = {Gm + m^^) A 1 for short. Let us first note that 
for any k £ {1, . . . ,m}, ^^P(k) < ^fc" is equivalent to ^^Gm{p{k / m)) > k/m" . We 
now distinguish two cases: 

- Step- up case: assume p(^x) > tx, that is, Gm{p{X/m)) < X/m. Let us prove 
that k/m = U{\/m,Gm)- Since U{£,G) is a fixed point of the function 
G o p, we have G) G {0, 1/m, . . . , m/m}. Hence, 

k/m = max{k/m G {0, . . . , X/m} : Gm{p{k/m)) > k/m} 
= max{ti G [0, A/m] : Gm(/)(n)) > u}, 
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and we can conclude. 



- Step-down case: assume p(^x) < ^A, that is, Gm{p{X/m)) > X/m. First 
assume that k < m and prove that {k + l)/m = U{\/m, ^'m)- On the one 
hand, 

(A; + l)/m = min{A;/m G {(A + l)/m, m/m} : Gm{p{k/m)) < k/m} 
= min{A;/m G {A/m, . . . , m/m} : Gmipik/m)) < k/m} 
= mm{k/m £ {X/m, . . . ,m/m} : G'^{p{k/m)) < k/m}, (28) 

because mGrn{p{k/m)) is an integer. On the other hand, since G'^{p{X/m)) > 
X/m, we have 

W(A/m, G^) = min |n G [A/m, 1] : G^(p(u)) < ii| 

= min |m G {A/m, . . . , m/m} : G'^{p{u)) <u^ , (29) 

because Z^(A/m, G^) G {0, 1/m, . . . ,m/m}. Combining pS] ) and ([29]) yields 
the result. Second, in the case where k = m, then for any k/m £ {X/m, . . . ,m/m}, 
we have Gm{p{k/m)) > k/m. Hence, for all k/m G {X/m,...,{m — 
l)/m}, G'„(p(/c/m)) > k/m which entails W(A/m, G'„J = 1. Hence, k/m = 
U{X/m,G'^) in that case. Finally, the inequality k/m < U{X/m,G'^) al- 
ways holds. 



7.3 Proof of Theorem \4A\ 



Let us first prove the result in FM(m,mo,-F) model. We recall Gm(x) := 
Yl^i ^{Pi ^ G^^ (x) := (1 — C) + Cx and we put u := k/m, where 
k is defined hj We can easily check from the definition ([9]) that if G, G' are 
two nondecreasing functions such that \/x G [0, 1], G{x) > G'{x), then U[X, G) > 



U{X, G'). Based on the bound (10), we deduce that \/5 G (0, 1), 



sup |G„(x) - G9^{x)\ <5- — \(l{u^<u<uj}. 

a:e[0,l] ^ I 
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Figure 6: Plot of Gm(yo(-)) for p{'^^ = 0-5a; and m = 10 p-values. The order A of the SUD 
procedure is displayed by the dashed line while the value of fc/m is displayed by the dotted 
line. Left: the SUD procedure of order A = 8 is such that fc/m — U{X/m,'Gm) — 0.7. 
Right: the SUD procedure of order A = 4 satisfies fc/m — 0.7 but U{\/ni, Gm) = 0.4. 



As a consequence, in the DU(m, A;) model, k G {mo — l,mo}, by using Gm{x) 
GW{x) = {k/m - C)(l - Gfc(x)) - C(Gfc(x) - x), we obtain 



ns{k) := { sup \Gk{x) -x\< C^{5 - u - —) \ C {uj < u < uj} . (30) 

1x6 [0,1] 



Remember that pi, . . . ,pmo correspond to true nulls, hence, when k £ {mo — 
l,mo}, Gfc involves only variables which are i.i.d. uniform. As a consequence. 



by using the DKW inequality with Massart's (1990) optimal constant, we have 



in the DU(m, A;) model and for k £ {mo — 1, mo}, 
^DU(m,k) msikyr] < 2 exp I -2k 



< 2 exp < —2m [ 6 — u 



m 



(1 - wcu 



(31) 



because k/m > C, — ^ and C ^ 1- 
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7.3.1 Upper bound 

Let q{x) = p{x)/x when x G (0, 1] and q{0) = lim^_j.Q+ p{x)/x (the hmit exists 
in M because x G (0,1] — )• p{x)/x is non-decreasing). Applying Theorem 4.3 of 



Finner et al. (2009), we have 



FDR(SUDA(t),mo,F) < — EDu(„-i)te(n)] 



m 



< — ^ + —rBV(m,mo-l)m5{mo " (32) 
m uT m \ ' •J / 



because q is non-decreasing, is positive and by (30). Next, by (31), we obtain 
the foUowing upper-bound: 

FDR(SUDA(t),mo,F) < + !I^2exp |-2m U-u-^Y {^-^/0+ 



(33) 



7.3.2 Lower bound 

In the model DU(m, mo) with rrio < m, we have u > a.s. and thus 
FDR(SUDA(t),mo,F = 1) = — EDu(m,mo) I r I 

> ^DU(m,mo) ^ l{ils{mo)} 



mo 

> JEDU(m,mo) , 



'-°^^("^"ill{f^,(mo)} 



> 



mo p{ug ) mo 1 



m Ur 



1 A^DU(m,mo) [(f^5("i0))1 , 



m 1 — C 



by (30) and because uJ >1 — (. From ^31n, we obtain the lower-bound 



FDR(SUDA(t),mo,F 



^ "^0 Pjus ) _ mo 1 
^) — ^ ™ 1 



m Ug ml — C I \ m 



1 



2exp<( -2m ( — j (l-u/C)^ 

(34) 



Finally, (33) and (34) yield the result. 
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7.3.3 Proof for random mixture model 



In the RM(m, ttq, F) model with ttq = C) the distribution of mo is binomial with 
parameters {m,Q. In particular, is random. However, we can write 

E[FDP(SUDA(t), mo)] < E [FDP(SUDA(t), mo)l{i/ < 7}] 

+ 2P(|mo/m-7ro| > 7- 1/m). (35) 

(36) 



Additionally, using Hoeffding's ( 1963 ) inequality, we can write 



'(|mo/m-7ro| > 7 - 1/m) < 2e-2™(T-i/™)^. 



Combining (35) and (36) with (12) finishes the proof. 



7.4 Proof of Corollary |4.5| 

First consider the FM(m, mo,-F) model with mo < m (the case rriQ = m is 
trivial). Let i^m = ^a^ke{mo-i,mo}{\^ - Cm|} and consider 5m G {i^m, 1) that 
satisfies for large m, 

2(1 - fm/Cm)('5m " ^ m " 1/m)^ = (log m)/m, (37) 
so that e-2™('5--'^--i/™)+(i-!^m/U)+ = 1/^ for large m. Since i^m < 2/m by 
assumption, we have 6m oc (log m)/m. From Theorem 
prove 



4.4 



it is sufficient to 



0{6m/{l-Cm)). 



From Section 4.2, this holds for the linear critical value function. This also 



holds for the AORC as soon as Xm/m < vs^, which is the case for large m by 
assumption. 

The proof in the RM(m, ttq, F) model is similar by taking additionally 7^ oc 
^(log m)/m such that 2e-2"^(T'"-i/'^)+ = 1/m. 
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A Formulas for FDP distribution 



From Theorem 5.2 and (25), we can compute the exact c.d.f. of the FDP of any 
SUD procedure in the fohowing way, for each fixed number m > 2 of hypotheses. 

Corollary A.l. Let A € {1, . . . ,m} and consider any threshold collection t. Fix 
an arbitrary x G (0, 1). Then the following holds: 

(i) in the model RM(m,7ro,F), for any ttq G [0, 1], F £ J^, we have 

A-l [xk] 
m [xk\ 

kjy, (38) 

k=X j=0 

(a) in the model FM{m,mQ,F), for any niQ G {0, . . . ,m}, F £ T, we have 

A-l moA[xA;J 

m A t 

k=0 j=OV(k-m+mo) 
m mo/\\xk\ 
+ Q.m,mo,F{{t\y tj)i<j<m,k,i). 

k=\ j=OV(fe— m+mo) 

(39) 

B FDR(SD) can exceed FDR(SU) in an extreme con- 
figuration 

Lemma B.l. Consider the FM(m,mo,-F) model with F{x) = l{x > 1} (i.e., 

all the p-values under the alternative are constantly equal to 1). Consider the 
threshold collection t defined by t^ = to, 1 < k < m — 1 and tm = ^, for some 
to G (0, 1). Then we have for any A G {1, . . . , m — 1}, 

FDRiSUDx{t)) = l-{l-tor^'; 
FDR{SU{t)) = mo/m. 

In particular, FDR{SD{t)) > FDR{SU{t)) for to > I - (l - mo/m)^/"^". 

The proof is straightforward and is left to the reader. As an illustration, for 
m = 10 and mo = 7, 1 - (1 - mo/m)^/™" ~ 0.158. 
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C DU is an LFC for the A;-FWER 



We state here for the sake of completeness a straightforward generahzation of 



Lemma 1 of Firmer and Gontscharuk (2009) (see also Lemma 2.2 of Gontscharuk 



2010 ) concerning the LFCs of multiple testing procedures under a class of type I 
criteria containing in particular the /c-FWER (but not the FDR, as pointed out in 
the introduction). This result should be considered as already known by experts 
in the field, although we failed to locate a precise reference for it. The setting 
considered assumes independence of p-values corresponding to true nulls, but is 
more general than the fixed mixture model, since the p-values corresponding to 
true null hypotheses are only assumed to be stochastically larger than a uniform 
variable on [0,1]; also, the values corresponding to alternatives are not assumed 
to be identically distributed nor independent. 

Lemma C.l. Let m> 1 and niQ E {0, . . . ,m} be fixed. Let p = (pi, . . . ,p„i) he 
a family p-values with distribution by P such that {pi)i<i<mo form an indepen- 
dent family of variables, each stochastically lower bounded by a uniform variable. 
Assume that 5 is a multiple testing procedure rejecting all hypotheses having p- 
value less than a data- dependent threshold t*{p). Let R be a type I error criterion 
taking the form 

R{P,6)=Ep^p[(t>{Vm{d{pm, 

where Vm is defined in ([T]) and cj) is a function from N to M. 

Assume the two following conditions are satisfied: 
(i) t* is a nonincreasing function of each p-value; 
(a) (f) is nondecreasing. 

Then it holds that 

R{P,6) < R{DV{m,mo),6), 

that is, DU(m,mo) is an LFC for 6 among the set of distributions satisfying the 
properties described above. 

Proof. Using (i) and (ii) together entails that p i— )■ (j){Vra{S{p))) is a nonincreasing 
function of each p-value. Denote po = (pi, • • • ,Pmo, 0, . . . , 0) the p-value family 
obtained by replacing pi by for i > mo , and Pq the distribution of po when p 
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has distribution P. Obviously we have 



Ep.p[(/'(K.(<5(p)))] < Ep^p[(/.(y^(5(po)))] = Ep^pJ(/.(Kn(,5(p)))] 



Now applying Lemma A. 11 as cited by Gontscharuk (2010), we obtain 



~DU(m,mo) 

and thus the conclusion. □ 

A straightforward (though less immediately interpretable) extension of this 
result to procedures that are not necessarily threshold-based is to replace as- 
sumption (i) by (i'): p i— )• Vm{S{p)) is a non increasing function of each p-value. 
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